Focus adjustment apparatus and method, and image capturing apparatus

ABSTRACT

A focus adjustment apparatus comprises a detection unit that detects one or more main subject candidates for a main subject from image signals based on a predetermined first condition, a selection unit that selects the main subject from the one or more main subject candidates, an acquisition unit that acquires operation information of an operation unit for moving a position of a focus lens, and a focus adjustment unit that controls the focus lens so as to focus on the selected main subject in a case where an autofocus mode is set and the operation unit is not operated. In a case where the operation unit is operated, the selection unit selects the main subject based on an in-focus position of the focus lens and fixes the selected main subject.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a focus adjustment apparatus andmethod, and image capturing apparatus.

Description of the Related Art

In the conventional autofocus (AF) control, there are cases in which auser has to specify an AF frame around a subject that the user wishes tofocus. However, for example, in a shoulder-carrying type camera or thelike used for collecting news materials or the like, it may be difficultto set an operating device for operating an AF frame at a position onthe camera where the operator can easily operate it.

Accordingly, Japanese Patent Laid-Open No. 2010-97167 discloses atechnique for automatically setting an AF frame around a subject thatwas focused through a manual focus (MF) operation performed by anoperator.

Further, Japanese Patent No. 5474528 discloses a technique for, in acase where a focused position is within a face detection area as aresult of a manual focus operation performed by an operator, performingautomatic tracking and AF by setting an AF frame around a face if theface is detected or around a subject other than a face if no face isdetected.

On the other hand, image processing that automatically detects aspecific subject from an image is known. For example, in a digitalcamera or a digital video camera, a person's face is particularlydetected from a captured image, and the detection result is used as acontrol target to optimize the focus state and exposure state. However,with general face detection technology, stable detection may not beperformed in a case where sufficient facial features are not obtained,such as in a case where a person faces backward or wears ornaments onthe face.

In consideration of the above, Japanese Patent Laid-Open No. 2017-126915suggests a method of detecting a person's upper body or head portionfrom an image and using an area of the person's upper body, the headportion, or a face area estimated from the detection results as acontrol target. With this method, the face area of the person can bestably specified even when the person faces backward or wears ornamentson the face.

However, skillful technique is required to focus on a subject by MFoperation, and it is difficult to move the focus lens smoothly to thein-focus position. Further, in recent years, the resolution has beenincreased to 4K resolution and further to 8K resolution due to theincrease in the number of pixels, and it has become very difficult toconfirm the in-focus position on the monitor provided on the camera.Therefore, the operation by the operator to perform the manual focusoperation until the face is substantially in focus as described in thetechnique described in Japanese Patent No. 5474528 may causedeterioration of moving image quality when the skill of the operator forthe manual focus operation is not sufficient or when blurring of animage occurs due to the resolution of the monitor.

On the other hand, in the method of tracking a subject using the facedetection technique described in Japanese Patent Laid-Open No.2017-126915, there is a problem that, when the user wants to focus on anarbitrary subject on the screen different from a tracking subject, it isnot possible to change the subject to be tracked.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the abovesituation, and enables automatic determination of a main subject to befocused on as well as flexible change of the main subject according tothe user's intention.

According to the present invention, provided is a focus adjustmentapparatus comprising: a detection unit that detects one or more mainsubject candidates which are candidates for a main subject from imagesignals repeatedly output from an image sensor that performsphotoelectric conversion on light that is incident through an imagingoptical system based on a predetermined first condition; a selectionunit that selects the main subject from the one or more main subjectcandidates; an acquisition unit that acquires operation information ofan operation unit for moving a position of a focus lens included in theimaging optical system; and a focus adjustment unit that controls thefocus lens so as to focus on the main subject selected by the selectionunit in a case where an autofocus mode is set and the operation unit isnot operated, wherein the selection unit selects the main subject basedon an in-focus position of the focus lens and fixes the selected mainsubject in a case where the operation unit is operated, and wherein eachunit is implemented by one or more processors, circuitry or acombination thereof.

Further, according to the present invention, provided is an imagecapturing apparatus comprising: an image sensor; and a focus adjustmentapparatus comprising: a detection unit that detects one or more mainsubject candidates which are candidates for a main subject from imagesignals repeatedly output from the image sensor that performsphotoelectric conversion on light that is incident through an imagingoptical system based on a predetermined first condition; a selectionunit that selects the main subject from the one or more main subjectcandidates; an acquisition unit that acquires operation information ofan operation unit for moving a position of a focus lens included in theimaging optical system; and a focus adjustment unit that controls thefocus lens so as to focus on the main subject selected by the selectionunit in a case where an autofocus mode is set and the operation unit isnot operated, wherein the selection unit selects the main subject basedon an in-focus position of the focus lens in a case where the operationunit is operated and fixes the selected main subject, and wherein eachunit is implemented by one or more processors, circuitry or acombination thereof.

Furthermore, according to the present invention, provided is a focusadjustment method comprising: detecting one or more main subjectcandidates which are candidates for a main subject from image signalsrepeatedly output from an image sensor that performs photoelectricconversion on light that is incident through an imaging optical systembased on a predetermined first condition; selecting the main subjectfrom the one or more main subject candidates; acquiring operationinformation of an operation unit for moving a position of a focus lensincluded in the imaging optical system; and controlling the focus lensso as to focus on the selected main subject in a case where an autofocusmode is set and the operation unit is not operated, wherein the mainsubject is selected based on an in-focus position of the focus lens andthe selected main subject is fixed in a case where the operation unit isoperated.

Further, according to the present invention, provided is anon-transitory computer-readable storage medium, the storage mediumstoring a program that is executable by the computer, wherein theprogram includes program code for causing the computer to function as afocus adjustment apparatus comprising: a detection unit that detects oneor more main subject candidates which are candidates for a main subjectfrom image signals repeatedly output from an image sensor that performsphotoelectric conversion on light that is incident through an imagingoptical system based on a predetermined first condition; a selectionunit that selects the main subject from the one or more main subjectcandidates; an acquisition unit that acquires operation information ofan operation unit for moving a position of a focus lens included in theimaging optical system; and a focus adjustment unit that controls thefocus lens so as to focus on the main subject selected by the selectionunit in a case where an autofocus mode is set and the operation unit isnot operated, wherein the selection unit selects the main subject basedon an in-focus position of the focus lens and fixes the selected mainsubject in a case where the operation unit is operated.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention, andtogether with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram showing a configuration of a digital videocamera system according to a first embodiment of the present invention.

FIGS. 2A and 2B show a flowchart of focus adjustment processingaccording to the first embodiment.

FIG. 3 is a flowchart of a parameter setting process according to thefirst embodiment.

FIGS. 4A and 4B are diagrams for explaining a focus detection framesetting process according to the first embodiment.

FIGS. 5A and 5B show a flowchart of a main subject candidatedetermination process according to the first embodiment.

FIG. 6 is a flowchart of a main subject changing process according tothe first embodiment.

FIG. 7 is a flowchart of an alternative subject identifying processaccording to the first embodiment.

FIG. 8 is a flowchart of a main subject lock-on determination processaccording to the first embodiment.

FIGS. 9A to 9D are conceptual diagrams showing examples of displayedframes showing a subject detected while MF operation is performed andwhile the MF operation is not performed according to the firstembodiment.

FIGS. 10A to 10C are conceptual diagrams showing examples of displayedframes showing a main subject and a main subject candidate while the MFoperation is performed and while the MF operation is not performedaccording to the first embodiment.

FIGS. 11A to 11I are conceptual diagrams showing examples of displayedframes showing a main subject candidate and a main subject according tothe first embodiment.

FIGS. 12A and 12B show a flowchart of a main subject candidatedetermination process according to a second embodiment.

FIG. 13 is a flowchart showing a main subject candidate reset timesetting process according to the second embodiment.

FIG. 14 is a flowchart showing a reset time adjustment process for amain subject according to the second embodiment.

FIG. 15 is a flowchart showing a reset time adjustment process for amain subject candidate according to the second embodiment.

FIGS. 16A to 16F are conceptual diagrams showing areas for adjusting areset time for a main subject and a reset time for a main subjectcandidate, and distances between subjects according to the secondembodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments will be described in detail with reference tothe attached drawings. Note, the following embodiments are not intendedto limit the scope of the claimed invention, and limitation is not madean invention that requires a combination of all features described inthe embodiments. Two or more of the multiple features described in theembodiments may be combined as appropriate. Furthermore, the samereference numerals are given to the same or similar configurations, andredundant description thereof is omitted.

First Embodiment

Hereinafter, a first embodiment of the present invention will bedescribed in detail. In the following embodiment, a digital video camerasystem will be described as an example of an image capturing apparatus,but the present invention is not limited to this.

FIG. 1 is a block diagram showing a configuration of a lensinterchangeable type digital video camera system according to anembodiment of the present invention. The camera system in the presentembodiment is composed of a lens unit 10 and a camera body 20, and alens control unit 106 that controls the operation of the entire lensunit 10 and a camera control unit 207 that controls the operation of theentire camera system exchange data. In the present embodiment, thecamera control unit 207 is composed of a processor such as a CPU and anMPU and storage media such as a memory. Further, the camera control unit207 may include an arithmetic circuit, and the arithmetic circuit mayexecute some arithmetic functions performed by the processor. In thepresent embodiment, a lens interchangeable type camera system in whichthe lens unit 10 is detachable from the camera body 20 will be describedas an example, but the present invention can also be applied to a cameraintegrated with a lens unit.

First, the configuration of the lens unit 10 will be described. The lensunit 10 has an imaging optical system including a fixed lens 101, a zoomlens 108, a diaphragm 102, and a focus lens 103. The diaphragm 102 isdriven by a diaphragm driving unit 104 and controls the amount of lightto be incident on an image sensor 201, which will be described later.The focus lens 103 is driven by a focus lens driving unit 105 to adjustthe focus. The zoom lens 108 is driven by a zoom lens driving unit 109to adjust the zoom magnification. A lens unit 10 without a zoom functionmay be used.

The diaphragm driving unit 104, focus lens driving unit 105, and zoomlens driving unit 109 are controlled by the lens control unit 106, andthe aperture amount of the diaphragm 102 and the positions of the focuslens 103 and the zoom lens 108 are determined. When the user performsoperations for changing aperture, focus, and zoom magnification via alens operation unit 107, the lens control unit 106 controls the lensunit 10 according to the user operation. Further, the lens control unit106 controls the diaphragm driving unit 104, the focus lens driving unit105, and the zoom lens driving unit 109 according to control commandsand control information, described later, received from the cameracontrol unit 207. Further, the lens control unit 106 transmits lensinformation (for example, MF operation information and information onthe imaging optical system) to the camera control unit 207.

Next, the configuration of the camera body 20 will be described. Thecamera body 20 is configured to be able to acquire an image signal andAF signals converted from the luminous flux that has passed through theimaging optical system of the lens unit 10. The image sensor 201 isconfigured with a CCD, a CMOS sensor, or the like. The luminous fluxthat has passed through the imaging optical system is incident on thelight receiving surface of the image sensor 201, and a formed subjectimage is converted by photodiodes into electric charge (photoelectricconversion) according to the amount of incident light. Based on thedrive pulses given from a timing generator 209 according to theinstruction from the camera control unit 207, the electric chargeaccumulated in each photodiode is read out from the image sensor 201 inthe form of voltage signals as an image signal and an AF signal.

The image signal and AF signal read out from the image sensor 201 areinput to a CDS/AGC circuit 202, and correlation double sampling forremoving reset noise, gain adjustment, and signal digitization areperformed. Then, the CDS/AGC circuit 202 outputs the obtained image datato a camera signal processing unit 203, a face area detection unit 210,and a head area detection unit 211, and outputs the AF data to an AFsignal processing unit 204.

The camera signal processing unit 203 transmits the image data outputfrom the CDS/AGC circuit 202 to a display unit 205. The display unit 205is a display device (display member) configured with an LCD, an organicEL, or the like, and displays an image based on image data. Further, inthe mode for recording an image, the image data is recorded in arecording unit 206.

The AF signal processing unit 204 calculates information on the focusstate and reliability by a known method based on the AF data output fromthe CDS/AGC circuit 202. As information on reliability, contrastinformation, there are saturation information, defect information, etc.,and, in the case of an image sensor 201 corresponding to the so-calledon-imaging plane phase difference method, degree of two-image matching,degree of two-image steepness, etc. Then, the calculated focus state andreliability information are output to the camera control unit 207.

The face area detection unit 210 performs a known face detection processon the image data output from the CDS/AGC circuit 202, and detects theface area of a person in the shooting screen. That is, the face areadetection unit 210 detects a predetermined subject (here, a face) fromthe image data. The obtained face detection result is transmitted to thecamera control unit 207. As the face detection process performed by theface area detection unit 210, for example, an area having an attributeof a color of skin is extracted using information of tone of each pixelrepresented by the image data, then the face is detected using thedegree of matching between the detected area with the contour plate ofthe face prepared in advance. Alternatively, there is a method ofperforming face detection by extracting facial feature points such aseyes, nose, and mouth using a well-known pattern recognition technique;however, the present invention is not limited by the method of facedetection processing, and any method may be used. Further, the subjectis not limited to the face, and may be a subject other than the face ofa person.

The head area detection unit 211 detects a head area which is an arearelated to the face area detected by the face area detection unit 210 asthe target subject area from the image. That is, the head area detectionunit 211 detects a predetermined subject (here, the head) related to thepredetermined subject detected by the face area detection unit 210 fromthe image data. In the head area detection in the present embodiment,the head area is detected from the image data based on the learningdata, and the detection result is transmitted to the camera control unit207.

The camera control unit 207 sends information to the AF signalprocessing unit 204 to set a focus detection area at a position so as toinclude the face area and the head area on the shooting screen based onthe face detection result of the face area detection unit 210 and thehead detection result of the head area detection unit 211. In the firstembodiment, when a face is detected, the focus detection area is setbased on the position and size of the face area. On the other hand, whenthe face is not detected and only the head area is detected, the focusdetection area is set based on the position and size of the head area.Note that when only the head area is detected, the position and size ofthe face area of the subject may be estimated from the head detectionresult, and the focus detection area may be set based on the estimatedposition and size of the face area.

In addition, the camera control unit 207 performs control by exchanginginformation with each constituent of the camera body 20, and in additionto the processes within the camera body 20, executes camera functionsinstructed by a user in response to the input from a camera operationunit 208. The camera functions instructed by the user include, forexample, power ON/OFF, setting change, start of recording, switching ofAF (autofocus)/MF (manual focus) control, confirmation of recordedimages, and so forth. In addition, as described above, the cameracontrol unit 207 exchanges information with the lens control unit 106 inthe lens unit 10, sends control commands and control information for theimaging optical system, and acquires the information of the lens unit10. In addition, the camera control unit 207 calculates the drive amountof the focus lens 103 so as to achieve the in-focus state using theinformation of lens unit 10 received from lens control unit 106 and thefocus state and reliability information from the AF signal processingunit 204.

Further, the camera control unit 207 includes a main subject candidatedetermination unit 220, a main subject lock-on unit 221 and a mainsubject candidate reset determination unit 222. The main subjectcandidate determination unit 220 determines whether or not a detectedsubject can be a candidate for the main subject based on the facedetection result obtained by the face area detection unit 210 and thehead detection result obtained by the head area detection unit 211.Then, among the subjects determined to be candidates for the mainsubject, the main subject is selected through the main subjectdetermination process described later, and a frame indicating the mainsubject and a frame/frames indicating the main subjectcandidate/candidates are superimposed on the captured image on thedisplay unit 205 or the like.

Next, focus adjustment processing in the first embodiment will bedescribed with reference to FIGS. 2A and 2B. The focus adjustmentprocessing in the first embodiment is characterized by control based onthe detection results of the face area and the head area by the facearea detection unit 210 and the head area detection unit 211. This focusadjustment processing is executed according to a computer program storedin the camera control unit 207. Further, the series of the focusadjustment processing described below assumes to be performed by asystem that performs the focus adjustment process in synchronizationwith an image shooting cycle, but the present invention is not limitedto this.

First, in step S201, the face area detection unit 210 performs aparameter setting process for subject detection to set parameters usedfor subject detection. The details of the parameter setting process willbe described later with reference to FIG. 3 . Then, the face area isdetected by the face area detection unit 210 and the head area isdetected by the head area detection unit 211 using the set parametersaccording to AF or MF. After that, the process proceeds to step S202.

Next, in step S202, the AF signal processing unit 204 performs a focusdetection frame setting process that sets a focus detection frame/framesfor the AF signal output from the CDS/AGC circuit 202, and detects afocus state for each focus detection frame. The focus detection framesetting process performed here will be described later with reference toFIGS. 4A and 4B. After that, the process proceeds to step S203.

In step S203, the face detection result by the face area detection unit210, the head detection result by the head area detection unit 211, anda tracking search result are acquired, subject information is generated,and the process proceeds to step S204. The subject information generatedhere includes information about detection ID, face detection result,head detection result, tracking search result, position and size ofsubject area, main subject candidate flag, main subject flag, mainsubject lock-on flag, and focus detection result.

The detection ID is assigned to a subject detected by the camera controlunit 207 based on the acquired face detection result and the headdetection result. At that time, the camera control unit 207 determinesthe face area and the head area, and assigns the same detection ID tothe face area and the head detection result determined to belong to thesame subject. Further, a tracking search process for determining whetheror not the same subject detected in the current frame is also detectedin the immediately previous frame is performed. This tracking searchprocess may be performed by using a known method such as templatematching or histogram matching using the image of the immediatelyprevious frame, for example. When it is determined as a result of thetracking search process that the same subject exists in the image of theimmediately previous frame, the same detection ID as the detection IDassigned to the subject in the image of the immediately previous frameis assigned to the subject detected in the current frame.

The face detection result is information on the position and size of theface area detected by the face area detection unit 210, and the headdetection result is information on the position and size of the headarea detected by the head area detection unit 211. Instead of the facedetection result and the head detection result, information indicatingwhether or not information about a detected face area and a detectedhead area exists and, if exists, a link for accessing that informationmay be retained.

The tracking search result indicates information on whether or not thesame subject was found in the image of the previous frame as a result ofthe tracking search process described above.

The subject area indicates information regarding the position and sizeof the subject determined based on the face detection result and thehead detection result.

The main subject candidate flag indicates information added in the mainsubject candidate determination process performed in step S204 describedlater. The main subject flag is information added in a main subjectchanging process performed in step S206 and a main subject lock-ondetermination process performed in step S208, which will be describedlater, and is set for only one subject. The main subject lock-on flag isinformation also added in the main subject changing process performed instep S206 and the main subject lock-on determination process performedin step S208, which will be described later, and is used when fixing themain subject flag. As for the information of these flags, when the samesubject is present in the image of the immediately previous frame, theinformation with the same detection ID is inherited in step S203.

The focus detection result shows the focus state detected in step S202.

In step S204, the camera control unit 207 performs the main subjectcandidate determination process in which the candidate/candidates forthe main subject are determined among the subject/subjects detected instep S203 and the main subject candidate flag is set. This process willbe described later with reference to FIGS. 5A and 5B. After that, theprocess proceeds to step S205.

In step S205, MF operation is currently performed or not is determined.Information about MF operation is acquired by the lens operation unit107 or the camera operation unit 208, and basically indicates one of thefollowing three cases.

(Case 1) Set to the MF Mode and Directly Operate the Lens Unit 10

By switching an AF/MF switch (not shown) attached to the lens unit 10 bythe lens operation unit 107, or by switching between AF mode (autofocusmode) and MF mode (manual focus mode), both modes are not shown, on acamera menu by the camera operation unit 208, the lens is set to a statein which MF operation is possible. In that state, the lens operationunit 107 is directly operated to drive the focus lens driving unit 105.

(Case 2) Set to the MF Mode and Operate the Lens Unit 10 from the CameraBody 20

The camera operation unit 208 switches to the MF mode (not shown) usingthe menu of the camera body 20, thereby setting the lens unit 10 to thestate in which MF operation is possible. In that state, the user setsthe driving information for manual operation, and the information issent to the lens control unit 106 via the camera control unit 207 todrive the focus lens driving unit 105.

(Case 3) Set to the AF Mode and Directly Operate the Lens Unit 10

While the lens unit 10 and camera body 20 are set to the AF mode, thelens operation unit 107 is directly operated to drive the focus lensdriving unit 105. In this state, AF operation is restored when the MFoperation ends.

If the MF operation is currently performed, the process proceeds to stepS206, and if the MF operation is not currently performed, the processproceeds to step S207.

In step S206, the main subject changing process is performed in whichthe subject that the user wants to shoot is specified from the subjectson which the main subject candidate flag is set and then the mainsubject flag is set on the specified subject. This process will bedescribed later with reference to FIG. 6 . After that, the processproceeds to step S211.

On the other hand, in step S207, it is determined whether or not asubject designation instruction is given by a user from the cameraoperation unit 208. If it is determined that the subject designationinstruction is given, the process proceeds to step S208, and if not, theprocess proceeds to step S209. As a subject designation method, forexample, a method of designating the subject on a touch panel or amethod of designating the subject by pressing a cross key whosedirection and pressed state are detected may be considered.

In step S208, a main subject lock-on process is performed by the mainsubject lock-on unit 221. The main subject lock-on process is a processof determining the main subject based on the user instruction regardlessof the result of the main subject candidates determined in step S204.This process will be described later with reference to FIG. 8 . Afterthat, the process proceeds to step S211.

In step S209, it is determined whether or not the main subject islocked-on by determining the presence or absence of the main subjectlock-on flag set by the process described later in step S208. If themain subject is not locked-on, the process proceeds to step S210, and ifthe main subject is locked-on, the process proceeds to step S211.

In step S210, the main subject is determined based on the position andsize of each main subject candidate determined in the main subjectcandidate determination process in step S204. As parameters fordetermining the main subject, not only the position and size, but alsothe reliability of the results of face detection and head detection,orientation of the face and head, and whether both face and head aredetected or only one of them is detected, and so forth, may be used.Further, the main subject may be determined from the main subjectcandidate/candidates by using a known algorithm.

In step S211, a frame display process of displaying a frame around eachsubject on which the main subject candidate flag is set in the displayunit 205 to notify the user of the main subject candidate/candidates isperformed. Specifically, for the main subject determined in S206, S208,and S210, a main subject frame indicating that the subject is the mainsubject is displayed around the face portion or the head portion. Inaddition, among the subjects on which the main subject candidate flag isset, a candidate frame indicating that the subject is the main subjectcandidate is displayed around the face portion or head portion of thesubject on which the main subject flag is not set. Further, when themain subject lock-on flag is set, only the main subject frame isdisplayed around the face portion or head portion of the main subject,and the candidate frame is not displayed around the face portion or headportion of the subject on which the main subject candidate flag is set.

In step S212, it is determined whether or not the MF mode is set or MFoperation is currently performed. If the MF mode is set or the MFoperation is currently performed, the process proceeds to step S213where a lens control process for sending a drive command for the focuslens 103 to the lens control unit 106 based on the focus detectionresult of the main subject on which the main subject flag is set. Ifthere is no subject on which the main subject flag is set, for example,a focus detection frame is set in the center of the screen, and the lenscontrol process is performed based on the focus detection result or thelike of the set focus detection frame. Since a conventional method maybe used for the focus detection method and the lens control process,detailed description thereof will be omitted here.

If the MF mode is set or the MF operation is currently performed, as theuser is operating the focus lens 103, so the process ends.

Next, the parameter setting process performed in step S201 will bedescribed with reference to FIG. 3 . The parameter setting process is aprocess for setting the parameters referred to by the face areadetection unit 210. In the present embodiment, different thresholds areset depending on whether the MF operation is currently performed or not,and while the MF operation is performed, control is made so that even asmall face or a face with low reliability is easily detected in order todetect the subject aimed at by the user.

First, in step S301, it is determined whether or not the MF operation iscurrently performed. The determination here is made by determiningwhether or not the operation according to any of the above-mentionedthree cases is performed. If the MF operation is currently performed,the process proceeds to step S302, and if not, the process proceeds tostep S305.

In step S302, a threshold FaceTh1 for the detectable minimum face sizeis set. For example, if the threshold FaceTh1 is set by the pixel sizeon the image sensor 201 and a face having a size of 15 pixels or more isdesired to be detected, “15” is set.

Next, in step S303, a threshold RelianceTh1 for the reliability of theface detection is set. For example, if the reliability of the facedetection result is expressed with 10 steps and it is desired to detecta face having a reliability of 3 or more, “3” is set.

Then, in step S304, a threshold AreaTh1 for the face detection area isset, and the process ends. For example, if it is desired to set an areawhere the face detection is to be performed to 90% of the screen, “90”is set.

On the other hand, if it is determined in step S301 that the MFoperation is not currently performed, the process proceeds to step S305and a threshold FaceTh2 for the detectable minimum face size is set. Forexample, if the threshold FaceTh2 is set by the pixel size on the imagesensor 201 and a face having a size of 25 pixels or more is desired tobe detected, “25” is set.

Next, in step S306, the threshold RelianceTh2 for the reliability of theface detection is set. For example, if the reliability of the facedetection result is expressed with 10 steps and it is desired to detecta face having a reliability of 6 or more, “6” is set.

Then, in step S307, a threshold AreaTh2 for the face detection area isset, and the process ends. For example, if it is desired to set an areawhere the face detection is to be performed to 80% of the screen, “80”is set.

The relationships between the above thresholds are FaceTh1<FaceTh2,RelianceTh1<RelianceTh2, and AreaTh1>AreaTh2, so that a face is detectedeasier while the MF operation is performed. As described above, whilethe MF operation is performed, the lower limit of the minimum face sizeis made smaller, the lower limit of the reliability of face detection ismade lower, and the face detection area is widened compared to a casewhere the MF operation is not performed.

The subject detection result obtained in a case where the parameters areset as described above will be described with reference to FIGS. 9A and9B. FIG. 9A shows a case where subjects 901 and 902 are included in thescreen while the MF operation is not performed. In this case, thesubject 901 is detected by the subject detection, but the subject 902 isnot detected because the face is smaller than the threshold FaceTh2 forthe minimum face size set in step S305. Further, when the subject 902 islocated in the periphery of the screen and is outside the thresholdAreaTh2 for the face detection area set in step S307, it is unlikely tobe the main subject in the first place, so it is restricted so that thesubject detection will not be performed in that area.

When the subject 901 is determined as the main subject by the process instep S208 or S210 based on such a subject detection result, a mainsubject frame 903 indicating that the subject 901 is the main subject isdisplayed in step S211. On the other hand, no frame is displayed aroundthe subject 902. Further, only the subject 901 is subjected to the focusdetection process.

FIG. 9B shows a case where the subjects are the same as those of FIG. 9Aand the MF operation is currently performed. While the MF operation isperformed, the user may intentionally perform a focus operation on thesubject 902. Therefore, by lowering the threshold for face detection toexpand the face detection area so that more face can be easily detected,both the subjects 901 and 902 are detected.

When the subject 901 is determined as the main subject by the process instep S206 based on such a subject detection result, the main subjectframe 903 indicating the main subject is displayed around the subject901 in step S211. Further, a candidate frame 904 is displayed around thesubject 902, and both the subjects 901 and 902 are subject to the focusdetection process. In this way, while the MF operation is performed, byadditionally using the subject that is not a candidate for the mainsubject while the MF operation is not performed as a target of the focusdetection process, changing of the main subject that reflects the user'sintention becomes possible.

Next, the focus detection frame setting process performed in step S202will be described with reference to FIGS. 4A and 4B. The focus detectionframe setting process is a process of setting focus detection frames ina mesh pattern on an imaging surface, and focus detection is performedbased on AF signals obtained from each frame.

In the present embodiment, the methods of setting the focus detectionframes differ depending on whether the MF operation is currentlyperformed or not. In step S202, if the MF operation according to any ofthe above-mentioned (Case 1) to (Case 3) is detected, it is determinedthat the MF operation is currently performed. Then, depending on thedetermination result, the focus detection frames corresponding to a casewhere the MF operation is currently performed or the focus detectionframes corresponding to a case where the MF operation is not currentlyperformed are set.

In FIGS. 4A and 4B, subjects 401, 402, and 403 show examples of subjectsdetected by the face area detection unit 210 and head area detectionunit 211 in step S202 using the parameters set in step S201. FIG. 4Ashows a method of setting the focus detection frames while the MFoperation is not performed, and focus detection frames 405, 406, and 407are set for the subjects 401, 402, and 403 according to the sizes of thesubjects. This setting method has the following merits.

Efficient focus detection process is possible by excluding unnecessaryparts other than the subject.

The size of the frame is changed according to the size of the subject,and a stable focus detection result is achieved since a large focusdetection frame is not set for a small face, which reduces perspectivecompetition within the frame.

In the example shown in FIG. 4A, the focus detection frame is set foreach subject for the sake of explanation, however, while the MFoperation is not performed, the focus is adjusted only on the subject towhich the main subject flag is set. Therefore, only one focus detectionframe may be set.

FIG. 4B shows a method of setting the focus detection frames in a casewhere the MF operation is currently performed. In a case where the MFoperation is currently performed, the mesh-shaped frame 408 is setaround the center of the screen regardless of the subject detectionresult. This setting method has the following merits.

The focus detection results of multiple subjects can be acquired atonce.

Even if the number of subjects detected by subject detection increases,the number of focus detection frames does not increase.

In this way, by changing the method for setting the focus detectionframes between the case where the MF operation is currently performedand the case where the MF operation is not currently performed, stablefocus adjustment becomes possible while the MF operation is notperformed. On the other hand, by efficiently acquiring the focusdetection results of a plurality of subjects while the MF operation isperformed, it is possible to quickly respond to the changing of the mainsubject.

Next, the main subject candidate determination process performed in stepS204 will be described with reference to FIGS. 5A and 5B. This processis performed by the camera control unit 207 for setting the main subjectcandidate flag on a subject that is a candidate for the main subjectamong the subjects detected in step S203. The main subject candidateflag is a flag that indicates a subject which may be determined as themain subject among the subjects found as face detection results.

First, among the subject information acquired in step S203, the presenceor absence of a face detection result is determined for one detection IDin step S501. If there is a face detection result, the process proceedsto step S502, and if there is no face detection result, the processproceeds to step S503. In step S502, the position and size of the facearea are stored as the information on the subject area. After that, theprocess proceeds to step S510.

In step S503, it is determined whether or not the MF operation iscurrently performed. If the MF operation is currently performed, theprocess proceeds to step S504, and if not, the process proceeds to stepS506.

In step S504, it is determined whether or not there is a head detectionresult. If there is a head detection result, the process proceeds tostep S505. On the other hand, if there is no head detection result, theprocess proceeds to step S512, the main subject candidate flag iscleared, and then the process proceeds to step S517.

In step S505, the position and size of the head area of the subject arestored as the information on the subject area. As described above, whilethe MF operation is performed, if no face is detected but if the headarea is detected, the information on the head area is stored as theinformation on the subject area. Then, the process proceeds to stepS510.

In step S506, it is determined whether or not the main subject candidateflag is set. Here, if the main subject candidate flag is set on asubject having the same detection ID as a subject in the previous frame,it is determined that the main subject candidate flag is set, andotherwise it is determined that the main subject candidate flag is notset. In a case where the main subject candidate flag is set, the processproceeds to step S505 and the position and size of the head area of thesubject are stored as the information on the subject area. In this way,if the main subject candidate flag has been set as the face area of thesame subject has been detected in the previous frame, or the head areais detected while the MF operation is performed, even if the face areais not detected in the current frame, the information on the head areais stored as the information on the subject area.

On the other hand, if the main subject candidate flag is not set, theprocess proceeds to step S512, and the main subject candidate flag iscleared, and then the process proceeds to step S517.

In step S510, the main subject candidate flag is set, and the processproceeds to step S513.

In step S513, it is determined whether or not the MF operation iscurrently performed, and if the MF is currently performed, the processproceeds to step S514, and if not, the process proceeds to step S516.

In step S514, the position in the focus detection frame of the subjectcurrently under process to which the main subject candidate flag is setis specified. In this process, it is specified in which area or areas ofthe focus detection frame 408, arranged in a mesh pattern as shown inFIG. 4B, the center position of the subject currently under process islocated. This is a process required to specify which of main subjectcandidates the user wishes to focus on while the MF operation isperformed. After that, the process proceeds to step S515.

In step S515, focus detection is performed based on the AF signals fromthe area or areas in the focus detection frame 408 corresponding to theposition specified in step S514, the focus detection result is stored,and the process proceeds to step S517.

On the other hand, in step S516, focus detection is performed based onthe AF signals from the subject area located at the position and havingthe size stored in any of steps S502, S505, and S509, and the focusdetection result is stored. Here, since the MF operation is notcurrently performed, the focus detection result corresponding to thefocus detection frame set for each subject, such as the focus detectionframes 405, 406, and 407 of FIG. 4A, is acquired. After that, theprocess proceeds to step S517.

In step S517, it is determined whether or not there is an unprocessedsubject (detection ID), and if the process of setting the main subjectcandidate flag and the process of storing the focus detection resulthave been completed for all the detection results, the process ends. Ifthere is an unprocessed subject (detection ID), the process returns tostep S501 and the above-described process is repeated.

As described above, in the present embodiment, the determination on theresult of head detection is changed depending on whether the MFoperation is currently performed or not. That is, while the MF operationis performed, in order to detect the subject aimed at by the user, aflag is set as a candidate for the main subject even on a small facewhose eyes and nose cannot be detected and a face that does not faceforward. On the other hand, while the MF operation is not performed, asmall face whose eyes and nose cannot be detected and a face that doesnot face forward are generally unlikely to be the main subject, so theflag is not set as not being a candidate for the main subject. However,if the face of the subject whose face had been detected is not detectedand the head detection result is valid, the main subject candidate flagis continuously set on the subject. This makes it easier to capture thesubject aimed at by the user while the MF operation is performed, andotherwise it is possible to perform focus adjustment control on the facethat is likely to be of the main subject.

An example of setting the main subject candidate flag as described abovewill be described with reference to FIGS. 9C and 9D.

FIG. 9C shows a case where subjects 905 and 906 are included in thescreen, while the MF operation is not performed. Since the face area ofthe subject 905 is detected, the main subject candidate flag is set. Onthe other hand, the subject 906 shows a case where the face area is notdetected but the head area is detected. In this case, if the mainsubject candidate flag is not set on the subject 906 appeared in theimmediately previous frame, the main subject candidate flag is also notset on the subject 906 in the current frame. This is because only thehead area of the subject 906 has been detected, and there is a lowpossibility that a subject in the state where the face has not beendetected in the past will be the main subject.

When the subject 905 is determined as the main subject in the process instep S208 or S210 based on the main subject candidate flag set in thisway, a main subject frame 907 indicating the main subject is displayedaround the subject 905 in step S211. On the other hand, no frame isdisplayed around the subject 906. Further, only the subject 905 issubjected to the focus detection process.

FIG. 9D shows the case where the same subjects 905 and 906 are includedin the screen while the MF operation is performed. Since the face areaof the subject 905 is detected, the main subject candidate flag is seton it. Further, although the face area of the subject 906 is notdetected, since the head area is detected, the main subject candidateflag is set on the subject 906.

In a case where the subject 905 is determined as the main subject by theprocess in step S206 based on the main subject candidate flag set inthis way, the main subject frame 907 indicating the main subject isdisplayed around the subject 905 in step S211. Further, a candidateframe 908 is displayed around the subject 906, and both the subjects 905and 906 are subjected to the focus detection process.

If the user intentionally performs a focus operation on the subject 906in this state, there is a possibility that the user wishes the subject906 to be the main subject. Therefore, even if the face is not detected,if there is a head detection result, the candidate frame 908 isdisplayed and the subject 906 is subjected to the focus detectionprocess. In this way, while the MF operation is performed, by subjectinga subject that is not a candidate for the main subject while the MFoperation is not performed to the focus detection process, it becomespossible to reflect the user's intention when changing the main subject.

Next, the main subject changing process performed in step S206 of FIG.2B will be described with reference to FIG. 6 . This process is executedwhile the MF operation is performed.

In step S601, it is determined whether or not there is one or moresubjects on which the main subject candidate flag is set. If there isone or more subjects on which the main subject candidate flag is set,the process proceeds to S604, and if there is none, the process proceedsto step S602.

In step S602, a main subject lock-on flag is cleared for all of thedetected subjects, and the process proceeds to step S603. In step S603,the main subject flag is cleared for all of the detected subjects, andthe main subject changing process is terminated. In this case, neitherthe main subject frame nor the candidate frame is displayed in stepS211.

On the other hand, in step S604, it is determined whether or not thereare two or more subjects on which the main subject candidate flag isset. At least two subjects are required to change the main subject.Therefore, if there are two or more subjects, the process proceeds tostep S606, and if not, the process proceeds to step S605.

In step S605, since there is only one subject on which the main subjectcandidate flag is set, the main subject flag is set on that subject, andthe main subject changing process is terminated.

In step S606, an alternative subject identifying process for identifyingthe changing of the main subject is performed based on the operationinformation of the lens unit 10 and the focus detection result, and theprocess proceeds to step S607. The process of step S606 will bedescribed later with reference to FIG. 7 .

In step S607, it is determined whether or not the main subject ischanged. If changed, the process proceeds to step S608, and if not, theprocess proceeds to step S610.

In step S608, the main subject lock-on flag is set on the changedsubject specified in step S606, and further, in step S609, the mainsubject flag is set on the changed subject specified in step S606, andthe main subject changing process is terminated.

In step S610, since the main subject is not changed, the main subjectflag is set on the subject having the same detection ID as the mainsubject determined in the previous routine, and the main subjectchanging process is terminated.

As described above, in a case where the main subject is changed whilethe MF operation is performed, by setting the lock-on flag on thechanged subject, it is possible to prevent the main subject from beingchanged when the AF frame is set after the MF operation is completed.This makes it possible to continuously focus on the subject intended bythe user during the MF operation as the main subject.

Next, the alternative subject identifying process performed in step S606will be described with reference to FIG. 7 .

First, in step S701, the focus detection result of the main subject onwhich main subject flag is set is acquired and held as DEF_Main, thenthe process proceeds to step S702.

In step S702, the depth of focus is calculated from the informationobtained from the lens control unit 106, and N times the depth of focusis set as a changing threshold DEF_Th. This changing threshold DEF_Th isa threshold value for determining whether the user wants to change thesubject when the amount of blur of the main subject becomes N times thedepth of focus or more. Note that this threshold value is an example,and may be another value as long as it can detect the change of the mainsubject.

In step S703, it is determined that whether or not there is a focusdetection result of another subject that can be a main subject bydetermining whether or not there is a focus detection result of asubject on which the main subject candidate flag is set but on which themain subject flag is not set. If there is a focus detection result, theprocess proceeds to step S704, and if there is no focus detectionresult, the process proceeds to step S706.

In step S704, the focus detection result DEF_Main of the main subject onwhich the main subject flag is set is compared with the changingthreshold DEF_Th. If DEF_Main>DEF_Th, it is determined that the user haschanged the main subject, and the process proceeds to step S705. If itis determined that the user has not changed the main subject, theprocess proceeds to step S712 where it is determined that the mainsubject is not changed, and the process ends.

In step S705, among the subject/subjects on which the main subjectcandidate flag is set, the subject with the smallest amount of blur isselected as a new main subject.

On the other hand, in step S706, the operation direction of the focusring is acquired, the information about the moving direction (toward theclosest or infinite end) of the focusing position of the focus lens 103that can be acquired from the lens control unit 106 is acquired, and theprocess proceeds to step S707. In step S707, the relative distance fromthe camera system to each subject on which the main subject candidateflag is set is determined from the size of the subject. Here, from thesize of the face, it is judged whether the subject is at a positioncloser to or farther from the camera system than the main subject.Basically, if the size of the subject is larger than the size of themain subject, the subject subject is determined on the closer side withrespect to the main subject, and if the size is smaller, the subject isdetermined to be on the farther side with respect to the main subject.After that, the process proceeds to step S708.

In step S708, the focus detection result DEF_Main of the main subject onwhich the main subject flag is set is compared with the changingthreshold DEF_Th. If DEF_Main>DEF_Th, it is determined that the user haschanged the main subject, and the process proceeds to step S709. If itis determined that the user has not changed the main subject, theprocess proceeds to step S712 where it is determined that the mainsubject is not changed, and the process ends.

In step S709, it is determined whether or not the operation direction ofthe lens is toward the closest end. If the operation direction acquiredin step S706 is toward the closest end, the process proceeds to stepS710, and if not, the process proceeds to step S711.

In step S710, the main subject is specified from the subject/subjects onthe closer side. This makes it possible to select the main subject fromthe subject or subjects whose sizes are small or for which the focusdetection results could not be obtained due to the shooting conditions.One of the subject/subjects that exists/exist in the operating directionof the focus lens is selected and specified as the main subject. As aspecifying method, as described above, a subject with a high possibilityof being the main subject is selected based on the size and position ofthe face.

In step S711, the main subject is specified from the subject/subjects onthe infinity side. Since the processing content is the same as that ofstep S710, the description is omitted. After that, the alternativesubject identifying process is terminated.

As a result, the focus detection result of the subject is acquiredduring the MF operation, and the change of the focus detection result ismonitored, so that the main subject can be automatically changed duringthe MF operation. Further, depending on the shooting conditions, thereis a case where the focus detection result may not be obtained. Even insuch a case, it is possible to determine the changing of the mainsubject based on the operating direction of the lens and the face size.

Here, the changing of the main subject when YES is determined in stepS704 will be described with reference to FIGS. 10A to 10C.

FIG. 10A shows an example of a display state before the MF operation,and shows a case where there are two subjects, a first subject 1001 (outof focus) and a second subject 1002 (in focus). Here, since the mainsubject flag is set on the second subject 1002, in step S211 of FIG. 2B,a main subject frame 1004 indicating that the second subject 1002 is themain subject is displayed around the second subject 1002. On the otherhand, since the main subject candidate flag is set on the first subject1001, a candidate frame 1003 indicating that the first subject 1001 is amain subject candidate is displayed.

FIG. 10B shows a case where the focus lens 103 is driven in thedirection toward a position at which the first subject 1001 will be infocus from the state in which the second subject 1002 is in focus by theMF operation. When the focus detection result DEF_Main of the secondsubject 1002 becomes larger than the changing threshold DEF_Th, the mainsubject flag is cleared. Therefore, in step S211, instead of the mainsubject frame 1004 indicating that the subject is the main subject, acandidate frame 1006 indicating that the subject is the main subjectcandidate is displayed. On the other hand, since the main subject flagis set on the first subject 1001, a main subject frame 1005 indicatingthat the first subject 1001 is the main subject is displayed.

FIG. 10C is a diagram for explaining the above state in chronologicalorder. In FIG. 10C, the vertical axis represents the focus detectionresult, and 0 represents an in-focus state. The horizontal axisrepresents time. Further, the focus detection result of the firstsubject 1001 is shown by a curve 1008, and the focus detection result ofthe second subject 1002 is shown by a curve 1007.

The state shown in FIG. 10A is a state based on the focus detectionresult at time t1, and by driving the focus lens 103 from that state,the focus detection result 1008 of the first subject 1001 decreases,while the focus detection result 1007 of the second subject 1002increases. At a point 1012 (time t2) when the focus detection result1007 exceeds the changing threshold DEF_Th, the camera determines thatthe user wants to focus on the first subject 1001, and changes the mainsubject (state shown in FIG. 10B).

In this way, by changing the main subject while driving the focus lens,it is possible to simplify the operation of the focus lens by the user.That is, by setting AF, if the user stops the operation of the focusring after the main subject frame 1005 indicating the main subject isdisplayed, AF control is performed on that subject, so the continuousfocus adjustment from the MF operation is possible.

Next, the main subject lock-on determination process performed in stepS208 of FIG. 2A will be described with reference to FIG. 8 .

In a case where a frame that indicates a face area of a person otherthan the main subject is displayed, if the frame is kept displayedaround an unintended subject, the frame may be a disturbance. However,there are some scenes where it is better to follow the subject for acertain period of time depending on the shooting situation. For example,in a case where there are multiple people to be shot on the screen, if asubject/subjects other than the main subject is/are followed as a mainsubject candidate/candidates for a certain period of time, when the mainsubject disappears and cannot be followed, by changing the main subjectto another subject, it is possible to prevent from focusing on oradjusting exposure on the background.

Therefore, in this process, while improving the tracking performance ofthe main subject in the situation where the main subject disappears, itis controlled to select an appropriate main subject according to theshooting situation.

First, in step S801, it is determined whether or not there is a subject,determined in step S204, on which the main subject candidate flag is setwithin a predetermined area from the position of the subject (forexample, the touch position) designated by the subject designationinstruction received from the camera operation unit 208 in step S207.The predetermined area described above may be the size of the face areaor head area stored in the subject area of the subject information, ormay be within a distance of, for example, twice the size of the facearea or head area. If there is a main subject candidate within thepredetermined area, the process proceeds to step S802, and if not, theprocess proceeds to step S809.

In step S802, it is determined whether or not there are a plurality ofmain subject candidates within the predetermined range of the touchposition, and if it is determined that there is only one person, theprocess proceeds to step S803, and if it is determined that there are aplurality of persons, the process proceeds to step S804.

In step S803, it is determined whether or not the reliability of theface area or head area determined to be within a predetermined area fromthe touch position and stored in the subject area of the main subjectcandidate is equal to or higher than a predetermined threshold. If it isdetermined that the reliability is equal to or higher than thethreshold, the process proceeds to step S805, and if it is determined tobe less than the threshold, the process proceeds to step S806.

On the other hand, in step S804, it is determined whether or not thesizes of the face areas or head areas of all the main subject candidatesdetermined to be within the predetermined area from the touch positionare equal to or less than the threshold. If it is determined that theall sizes are equal to or less than the threshold, the process proceedsto step S807, and if it is determined that there is a subject having asize larger than the threshold, the process proceeds to step S808.

Next, in steps S805 and S808, the main subject candidate within the areafrom the touch position and closest to the touch position is locked-on.

On the other hand, in steps S806, S807, and S809, the features of asubject is extracted based on the touch position to track an objectother than the face or head of a person (hereinafter, referred to as“object tracking”).

Next, in step S810, the locked-on main subject candidate is determinedas the main subject, and the main subject candidate flag is set on thelocked-on main subject. Then, in step S811, the main subject candidateflag set on the subject/subjects other than the locked-on main subjectis cleared so that the subject other than the locked-on main subjectdoes not become the main subject candidate, then the process isterminated.

The reason for performing object tracking in step S806 when it isdetermined in step S803 that the reliability is less than the thresholdis that the reliability of head detection is low, for example, when apart of the head is out of frame. In such a case, the detected positionand size become unstable, and AF may become unstable. Therefore, if thereliability of the detected face area or head area is low, an area isspecified by tracking using the color information and brightnessinformation at the time of touch, and the subject at the positionspecified by the user is temporarily tracked as an object. Then, bychanging the tracking target to a face or head when a face or a highlyreliable head is detected nearby during tracking, it is possible toreduce a situation in which the AF becomes unstable.

Further, the reason for performing object tracking in a case where it isdetermined in step S804 that there are a plurality of faces and heads asdetection results within a predetermined area from the touch position,is that there is a possibility that an unintended person will be trackedamong those people due to the deviation of the touch position or thelike specified by the user. Therefore, the subject at the positionspecified by the user is temporarily tracked as an object, and then,when a face or head of a predetermined size or larger is detected withinthe predetermined area, the tracking target is changed to the face orhead. In this way, it is possible to reduce the cases of focusing on anunintended person.

The above-described process will be explained with reference to FIGS.11A to 11I.

FIG. 11A shows a scene in which one person 1101 enters a screen withfacing backwards when the MF operation is not performed. In this case,since the main subject candidate flag is cleared for the person 1101 instep S512 of FIG. 5B, no frame is displayed.

FIG. 11B shows a case where the same person 1101 faces the front,changed from the state shown in FIG. 11A. In this case, since the mainsubject candidate flag is set in step S510 of FIG. 5B and there is onlyone main subject candidate, the person 1101 is automatically selected asthe main subject in step S605 of FIG. 6 and a main subject frame 1102 isdisplayed.

FIG. 11C shows a case where the person 1101 faces backwards, changedfrom the state in which the person 1101 faces the front shown in FIG.11B. In this case, since the person 1101 has faced the front once, themain subject candidate flag has been set and the head area has beendetected, so that the main subject candidate flag is set in step S510 ofFIG. 5B. Further, in step S605 of FIG. 6 , the person 1101 isautomatically selected as the main subject, so the main subject frame1102 continues to be displayed. By keeping the person 1101 as the mainsubject candidate, it is possible to continue focusing on the person1101 and prevent from focusing on or adjusting brightness on anunintended area.

FIG. 11D shows a scene in which there are a plurality of people (person1103 and person 1105) when the MF operation is not performed. In thiscase, the main subject candidate flag is set on the person 1103 facingthe front in step S510 of FIG. 5B, and since there is only one mainsubject candidate, the person 1103 is automatically selected as the mainsubject in step S605 of FIG. 6 , and a main subject frame 1104 isdisplayed. On the other hand, since the person 1105 is facing backwards,the main subject candidate flag is cleared in step S512, and no frame isdisplayed.

FIG. 11E is a case where the person 1105 faces the front, changed fromthe state shown in FIG. 11D. In this case, the face area of the person1105 is detected, the main subject candidate flag is set in step S510 ofFIG. 5B, and a candidate frame 1106 is displayed. In this case, theperson 1105 can also be the main subject, but if there is no instructionto change the main subject by MF operation (step S206), by subjectdesignation instruction (step S208), or by automatic judgment (stepS210), the present person 1103 who has already been selected as the mainsubject continues to be the main subject for the time being.

FIG. 11F shows a case where the person 1105 faces backwards, changedfrom the state in which the person 1105 faces the front as shown in FIG.11E. In this case, since the person 1105 has faced the front once, themain subject candidate flag has been set, and the head area has beendetected, the main subject candidate flag is continuously set in stepS510 of FIG. 5B. By continuously setting the person 1105 as the mainsubject candidate in this way, the person 1105 can be immediately set asthe main subject when the person 1103 is framed out.

FIGS. 11G to 11I show a case where there are a plurality of people(person 1103 and person 1105), and the user designates the person 1105by touching the person 1105 in the states shown in FIGS. 11D to 11F,respectively.

In the case of FIG. 11G, since the person 1105 is not detected as asubject, the person 1103 closest to the touch position is continuouslyselected as the main subject, and the main subject frame 1104 iscontinuously displayed.

Further, in the cases of FIGS. 11H and 11I, if the reliability of theface area or the head area is high, the person 1105 is locked-on as themain subject area by the processing of steps S801, S802, S803, S805,S810, and S811, and the person 1105 is selected as the main subject. Inaddition, a main subject frame 1109 is displayed. Further, since thesubject candidate flag set on the person 1103 is cleared in step S811,the main subject frame 1104 displayed around the person 1103 is deleted.

As described above, when a person facing backwards is touched, if thereis a head detection result at the touched position, the subject islocked-on using the head detection result regardless of whether it isalready set as the main subject candidate area or not. If the usertouches a person who is not yet set as the main subject area and mainsubject candidate area, and if there is a head detection result at theposition specified by the touch, by locking-on the area specified by thehead detection result as the main subject area using the head detectionresult, it will become possible to perform tracking on the main subjectregardless of changes in the shape and brightness the subject.

As described above, according to the first embodiment, it is possible tochange the subject to be focused on before driving the lens to thein-focus position. Furthermore, when AF is set, a highly versatile AF isrealized by selecting a representative subject as the main subject. Inaddition, by changing the selection method of the main subject triggeredby the MF operation, the subject aimed at by the user can be selected asthe main subject, which enables performance of AF according to theuser's purpose. Then, it is possible to continuously focus on thesubject selected by the MF operation as an AF target even after the MFoperation. Further, it becomes possible to easily change the mainsubject.

Further, the focus detection device according to the first embodimentcan keep focusing on a subject in a wide area by changing focusdetection area used for focusing operation according to the position ofthe subject in an area of a preset AF frame. Further, when it focuses ona subject other than the main subject, such as background (hereinafterreferred to as “miscellaneous subject”), the present invention canprevent it from being difficult to focus on a new main subject appearingon the closest side.

Second Embodiment

Next, a second embodiment of the present invention will be described.

The method described in the first embodiment for determining the mainsubject is very useful in a case where, for example, the operator hasdetermined a person to be shot and wishes to continuously focus on oradjust exposure on the person even when the face of the person cannot beseen. However, when shooting is performed basically on a person as themain subject, if the user wishes to focus on and adjust exposure onsubjects other than a person or landscapes without changing the shootingmode, tracking a person even if the face cannot be seen disables tofocus on and adjust exposure on subjects other than a person or onlandscape as long as there is a person on the screen.

In addition, when a subject was excluded from the main subjectcandidates immediately after the face was disappeared, there were thefollowing problems. That is, if an unintended person other than theperson who the operator is trying to shoot enters the screen and thenstays on the screen, the unintended person will not be excluded from themain subject candidates for a long time, and the unintended person maybe selected as the main subject, and the focus state and exposure maychange.

Therefore, in the first embodiment described above, even if the face ofa subject cannot be detected, but if the head area can be detected, thesubject is kept tracked as the main subject candidate, and the mainsubject is determined according to the designated method, and the focusadjustment and the exposure adjustment are performed. In situationswhere the face temporarily disappears as the main subject facesbackwards when the main target of shooting is a person, it will bedesired to keep the main subject as the main subject candidate as longas possible. However, in situations where the main target of shooting isnot limited to a person or where a plurality of people are to be shot,sometimes the operator may wish to keep the main subject as the mainsubject candidate and other times the operator may not wish to keep themain subject as the main subject candidate depending on the shootingsituation.

Accordingly, in the second embodiment, in addition to the firstembodiment, the time to keep a subject as the main subject candidate isset for each main subject candidate, and the time is changed accordingto the situation of the subject including the main subject. Hereinafter,the processing performed in the second embodiment will be described withreference to FIGS. 12 to 16F. Since the digital video camera systemshown in FIG. 1 is also used in the second embodiment, the descriptionthereof will be omitted.

FIGS. 12A and 12B show a flowchart showing a main subject candidatedetermination process in the second embodiment, which is performedinstead of the process shown in FIGS. 5A and 5B. In this process,processes in steps S1201 to S1203 for setting the time since detectionof a face area which has been detected is failed until a main subjectcandidate flag is to be cleared for each subject, which is a feature ofthe second embodiment, are added to the main subject candidatedetermination process in the first embodiment described above withreference to FIGS. 5A and 5B. Since the other processes are the same asthose shown in FIGS. 5A and 5B, the same step numbers are assigned andthe description thereof is omitted.

In step S1201, a reset time RT (>0) from when detection of a face whichhas been detected is failed until the main subject candidate flag is tobe turned off is set. The reset time RT shall be set for each subjectaccording to preset conditions, and the details will be described laterwith reference to FIG. 13 .

In step S1202, if the detection of the face area is failed, the resettime RT set in step S1201 is decremented. In step S1203, the mainsubject candidate reset determination unit 222 determines whether or notthe reset time RT is greater than 0, and if yes, the main subjectcandidate determination unit 220 sets the main subject candidate flag instep S511. On the other hand, if the reset time RT is 0 or less, it isassumed that the time until the main subject candidate flag set in stepS1201 is to be cleared has passed, and the main subject candidate flagis cleared in step S512.

With the processes of steps S1201 to S1203, the reset time RT accordingto the conditions is always set while the face is detected, and if thesituation where the detection of the face is failed continues for thereset time RT, the face can be excluded from the main subjectcandidates. This is because there are many situations and scenes inwhich a person whose face cannot be seen for a long time is not suitablefor focus adjustment control or exposure control, that is, not suitableas the main subject in general moving image shooting, and it isconsidered better to exclude it from the main subject candidates after acertain period of time has elapsed. However, if there is only one personto shoot, or if there are a plurality of people near the center of thescreen, the operator may want to continue selecting the person/people asthe main subject even if the face/faces cannot been seen. Therefore, thereset time RT is adjusted according to the situation.

FIG. 13 is a flowchart showing details of the setting process of thereset time RT for a main subject candidate performed in step S1201 ofFIG. 12A. In step S1301, it is determined whether or not the number n ofthe detected subjects is 1, and if it is 1, the process proceeds to stepS1302, and if it is more than 1, the process proceeds to step S1303. Instep S1302, the value of MainTimer1 is set as the reset time RT. Here,since the value of MainTimer1 represents a time for the situation whereonly the main subject is in the shooting screen, it is preferable to setMainTimer1 to a relatively long time, and in the present embodiment, itis set to 10 seconds as an example.

In step S1303, it is determined whether or not the subject beingprocessed is selected as the main subject in the main subjectdetermination process among the plurality of detected subjects, and ifit is selected as the main subject, the process proceeds to step S1304,and if not, the process proceeds to step S1305. In step S1304, the valueof MainTimer2 is set as the reset time RT. Here, the value of MainTimer2represents a time to continue setting the subject being processed as themain subject in the situation where a plurality of people exist in theshooting screen. Therefore, comparing to a case where only one personexists in the shooting screen, there is a possibility that another mainsubject candidate may be selected as the main subject, and thus ashorter time or the same time as in a case where only one person existsin the shooting screen may be set. In this embodiment, as an example, itis assumed that MainTimer2 is set to 10 seconds similarly to MainTimer1.

In step S1305, it is determined whether or not the number n of thedetected subjects is less than a threshold NumTh, and if the number n isless than the threshold NumTh, the process proceeds to step S1306, andif it is the same or more, the process proceeds to step S1307. In stepsS1306 and S1307, the time for following the main subject candidate isset, and it is set shorter than MainTimer1 and MainTimer2, the time forfollowing the main subject. This is because the main subject candidateis less likely to be a target to be shot than the main subject, and itis possible to prevent the main subject from being changedunintentionally by not setting the subject as the main subject candidatefor a long time. In step S1306, the value of SubTimer1 is set as thereset time RT. Here, the value of SubTimer1 represents a time to keepthe main subject candidate when the number of people is relativelysmall, and in the present embodiment, it will be described that thereset time RT is set to 3 seconds as an example.

On the other hand, in step S1307, the value of SubTimer2 is set as thereset time RT. Here, the value of SubTimer2 represents a time to keepthe main subject candidate when there are many people in the shootingscreen, and it is highly possible that subjects not intended by the userare in the screen, so the time shorter than SubTimer1 is set. In thepresent embodiment, as an example, it will be described that the resettime RT is set to 1 second.

In step S1308, a process of further adjusting, e.g., lengthening orshortening, the time for keeping a subject as the main subject set instep S1302 or S1304 is performed according to the conditions. In stepS1309, similarly to step S1308, a process of adjusting the time forkeeping a subject as the main subject candidate set in step S1306 orS1307 is performed according to the conditions. After finishing stepS1308 or S1309, the process returns to the process of FIG. 12A.

Next, the details of the process in step S1308 will be described withreference to FIG. 14 .

FIG. 14 is a flowchart showing a process of adjusting the time fortracking the main subject according to the conditions. In step S1401, anarea used for determining whether or not the main subject is in apredetermined area on the screen performed in step S1403 describedlater, is set. The concept of the predetermined area will be describedlater with reference to FIGS. 16A to 16F.

In step S1402, a threshold SizeTh for the size used for determiningwhether or not the sizes of the face area and head area of the mainsubject are large to some extent in step S1404, is set. This is becausewhen a person is shot as the main subject, the shot face tends to belarge to some extent, and it is highly likely that the small face is notof the person intended by the user, that is, of the main subject.

In step S1403, it is determined whether or not the main subject existsin the predetermined area set in step S1401. If it is determined thatthe main subject exists, the process proceeds to step S1404, and if itis determined that the main subject does not exist, the process ends.

In step S1404, it is determined whether or not the sizes of the facearea and the head area are larger than the threshold SizeTh for the sizeset in step S1402.

In step S1405, the reset time RT, set in steps S1302 and S1304 of FIG.13 , representing a time before tracking operation on the main subjectis stopped is set to a time longer than that of MainTimer1 andMainTimer2, and the process ends.

Here, the predetermined area will be described with reference to FIGS.16A to 16F. FIGS. 16A to 16F are schematic views showing a scene inwhich a person is a subject in the shooting screen. FIGS. 16A to 16Cshow a case where there is only one subject, that is, only the mainsubject is shot, and the face of the main subject disappears as timeelapses. FIGS. 16D to 16F show cases where there are a plurality ofsubjects.

When shooting a moving image with a person as the main subject, thesubject is generally framed near the center of the screen. Therefore, ifthe subject is detected near the center of the screen, it is highlylikely that the subject is the person that the user intends to shot.Therefore, it is determined whether or not there is a face area and ahead area of the detected subject in an area 1601 shown by a broken linein FIGS. 16A to 16F, and if there is any, the face area is continuouslytracked for a longer time after the face has disappeared. That is, thereset time RT, set in steps S1302 and S1304 of FIG. 13 , representing atime before tracking operation on the main subject is stopped is resetto a time longer than that of MainTimer1 and MainTimer2.

In the present embodiment, 15 seconds will be set as an example, but thepresent invention is not limited to this, and may be set to, forexample, 30 seconds or infinity (tracking is continued until bothdetection and tracking become impossible). Alternatively, MainTimer1 andMainTimer2 may be maintained. Note that FIG. 16A shows a case where theface area can be seen, and FIG. 16B shows a case where the main subjectfaces backwards and the face area cannot be seen. Since, the face areais within the area 1601 in both cases, the reset time RT until thesubject tracking is reset is 15 seconds. However, if the subject is nearthe boundary of the area 1601 and frequently enters and exits the area1601, the reset time RT may be set to 10 seconds sometimes, and set to15 seconds at other times, which makes the rest time RT unstable.Accordingly, once the main subject enters the area 1601, the area 1601may be expanded to an area 1602 as shown in FIG. 16C, and the reset timeRT may be maintained until the main subject goes out of the area 1602.By doing so, it is possible to prevent the reset time RT from becomingunstable then the subject exists near the boundary of the area.

Further, in the process shown in FIG. 14 , the conditions for extendingthe MainTimer has been described, but if the conditions imply that it isbetter not to track the main subject for a long time, the reset timebefore the tracking of the main subject is stopped may be shortened. Forexample, if the panning of the camera is detected, it may be determinedthat it is intended to change the main subject, and the reset time maybe shortened so as to stop tracking the subject whose face cannot beseen as the main subject.

Further, in the present embodiment, a rectangular area having apredetermined size and centered on the center of the screen is set asthe area 1601, but considering that there is a high possibility that themain subject exists in the area at the upper part of the screen, thecenter position of the area 1601 may be shifted upward or expandedupward. Further, the area may be changed according to the orientation ofthe camera. Further, in the present embodiment, the reset time RT isadjusted depending on whether or not the subject is in the area, but thepresent invention is not limited to this, and the reset time RT may bechanged according to the image height.

Next, the details of the process in step S1309 will be described withreference to FIG. 15

FIG. 15 is a flowchart showing a process of adjusting the time fortracking the main subject candidate according to the conditions. In stepS1501, an area used for determining whether or not the main subjectcandidate is in a predetermined area on the screen performed in stepS1503 described later, is set. In the present embodiment, this area willbe described as the same area as the area 1601 for the main subject ofFIGS. 16A to 16C, but it may be set smaller or larger than the area1601.

In step S1502, the difference D between the detected positions of thesubject area selected as the main subject and each main subjectcandidate is calculated. Here, in order to determine whether eachdetected subject is close to the main object or not in step S1504, thedistances between the center positions of the face areas and the headareas on the screen between the main subject and each main subjectcandidate are calculated as shown in FIGS. 16D and 16E.

In step S1503, it is determined whether or not the main subjectcandidate exists in the predetermined area set in step S1501, if it isdetermined that the main subject candidate exists, the process proceedsto step S1504, and if it is determined that the main subject candidatedoes not exist, the process proceeds to step S1505.

In step S1504, it is determined whether or not the difference D betweenthe detected positions of the main subject and each main subjectcandidate calculated in step S1502 is equal to or less than apredetermined value DTh. If it is equal to or less than the value DTh,it is determined that the distance between the main subject and the mainsubject candidate is short, and the process proceeds to step S1507. Onthe other hand, if the difference D is greater than the value DTh, it isdetermined that the distance between the main subject and the mainsubject candidate is long, and the process proceeds to step S1505. Thepredetermined value DTh may be a predetermined fixed value, or may becalculated based on the sizes of the face area and the head area.

In step S1507, the reset time RT, set in steps S1306 and S1307 of FIG.13 , representing a time before tracking operation of the main subjectcandidate is stopped is set to a time longer than that of SubTimer1 andSubTimer2, and the process ends.

As described above, if the subject is detected near the center of thescreen, it is highly likely that the subject is the person the userintends to shoot. Also, if the subject is a main subject candidate thatexists near the main subject, when the main subject is lost, it ishighly likely that the subject will be a candidate for the next mainsubject. Therefore, it is considered that a subject that meets theseconditions should be tracked for a longer period of time even if theface disappears. In the present embodiment, 5 seconds will be set as anexample, but the time is not limited to this, and SubTimer1 andSubTimer2 may be maintained.

Note that FIG. 16D shows a case where there are one subject whose facearea cannot be seen near the main subject whose face area can be seen inthe area 1601 and one subject whose face area cannot be seen outside thearea 1601. Further, FIG. 16E shows a case where there is one subjectwhose face area cannot be seen in the area 1601, and is away from themain subject. In the case of FIG. 16D, since the subject whose face areacannot be seen is within the predetermined area 1601 and near the mainsubject, the reset time RT until the subject tracking is stopped ischanged to 5 seconds, but in the case of FIG. 16E, since the subjectwhose face area cannot be seen is far from the main subject, the resettime RT until the subject tracking is stopped is not extended andremains as 3 seconds.

Further, FIG. 16F shows a scene in which many people are on the screen.When there are many subjects, since the main subject may be easilychanged against the user's intention, it is better not to track the mainsubject candidates too long. This corresponds to the case where thenumber of detected subjects is larger than the predetermined numberNumTh in step S1305 of FIG. 13 and a short time is set in step S1307.

In step S1505, it is determined whether or not the subject has beenselected as the main subject in the past, and if it has been selected,there is a high possibility that the user intends to shoot the subject,so the process proceeds to step S1507 and the time before the subjecttracking is stopped is adjusted, and if it has never been selected, theprocess proceeds to step S1506.

In step S1506, it is determined whether or not the time when the facehas been detected is longer than the predetermined time, and if it islonger, there is a high possibility that the subject is intentionallyframed in the screen by the user, and it is considered that it is betterto track the subject as the main subject candidate to some extent.Therefore, the process proceeds to step S1507, and the time before thesubject tracking is stopped is adjusted. On the other hand, if it isshort, there is a high possibility that the person is not intended bythe user, such as when the subject appears on the screen for a momentand turns to the back, so the process ends without adjusting the time.

Although the conditions for extending the Sublimer have been describedin the flowchart of FIG. 15 , if the conditions imply that it is betternot to track the main subject candidate for a long time as in the caseof the main subject, the reset time before the subject tracking isstopped may be shortened. For example, if the panning of the camera isdetected, it may be determined that it is intended to change the mainsubject, and the reset time may be shortened so as to stop tracking thesubject whose face cannot be seen as the main subject candidate.

As described above, according to the second embodiment, the time sincedetection of a face which has been detected is failed until the trackingof the subject is stopped is set for each subject. In particular, themain subject should be tracked for a longer time, and the other subjectsshould be tracked for a shorter time than the main subject. This makesit possible to improve the tracking performance of the main subject in asituation where the face cannot be seen, and to realize an appropriateselection of the main subject according to the shooting situation.

In the above-described embodiments, the detection results of the facearea and the head area of the person have been described, but thecombination of the face area and the body area of the person, thecombination of the face area and the head area or the body area of ananimal, and the like may be used.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2020-157676, filed on Sep. 18, 2020 which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A focus adjustment apparatus comprising: adetection unit that detects one or more main subject candidates whichare candidates for a main subject from image signals repeatedly outputfrom an image sensor that performs photoelectric conversion on lightthat is incident through an imaging optical system based on apredetermined first condition; a selection unit that selects the mainsubject from the one or more main subject candidates; an acquisitionunit that acquires operation information of an operation unit for movinga position of a focus lens included in the imaging optical system; and afocus adjustment unit that controls the focus lens so as to focus on themain subject selected by the selection unit in a case where an autofocusmode is set and the operation unit is not operated, wherein theselection unit selects the main subject based on an in-focus position ofthe focus lens and fixes the selected main subject in a case where theoperation unit is operated, and wherein each unit is implemented by oneor more processors, circuitry or a combination thereof.
 2. The focusadjustment apparatus according to claim 1, wherein the selection unitselects the main subject candidate located at a position nearest to thein-focus position of the focus lens as the main subject in a case wherethe operation unit is operated.
 3. The focus adjustment apparatusaccording to claim 1, wherein the selection unit reselects the mainsubject in a case where the operation unit is operated and a focus stateof the selected main subject is lower than a focus state set on thebasis of a depth of focus of the focus lens.
 4. The focus adjustmentapparatus according to claim 3, wherein the acquisition unit acquires amoving direction of the in-focus position of the focus lens by operationof the operation unit, and the selection unit reselects the main subjectfrom the one or more main subject candidates that exist at positions inthe moving direction of the in-focus position.
 5. The focus adjustmentapparatus according to claim 1, wherein, in a case where the operationunit is not operated and no main subject is fixed, the selection unitdetermines the main subject based on a predetermined second condition.6. The focus adjustment apparatus according to claim 1, wherein thedetection unit includes a first detection unit that detects a first areahaving a first feature from the image signal; and a second detectionunit that detects a second area which relates to the first area and hasa second feature from the image signal, and the detection unit detectsthe one or more main subject candidates based on information of thefirst area and the second area.
 7. The focus adjustment apparatusaccording to claim 6, wherein the first detection unit detects the firstarea using a third condition which differs between a case where theoperation unit is operated and a case where the operation unit is notoperated.
 8. The focus adjustment apparatus according to claim 7,wherein the third condition includes a size whose smallest limit of anarea to be detected as the first area is smaller in a case where theoperation unit is operated than in a case where the operation unit isnot operated.
 9. The focus adjustment apparatus according to claim 7,wherein the third condition includes an area of an image where the firstarea is to be detected, and the first detection unit detects the firstarea within the area larger in a case where the operation unit isoperated than in a case where the operation unit is not operated. 10.The focus adjustment apparatus according to claim 7, wherein the thirdcondition includes reliability whose lowest limit of the area to bedetected as the first area is made lower in a case where the operationunit is operated than in a case where the operation unit is notoperated.
 11. The focus adjustment apparatus according to claim 6,wherein the first condition is different between in a case where theoperation unit is operated and in a case where the operation unit is notoperated.
 12. The focus adjustment apparatus according to claim 11,wherein, in a case where the operation unit is operated, the detectionunit detects one or more subjects included in at least the first area orthe second area as the main subject candidates and, in a case where theoperation unit is not operated, the detection unit detects one or moresubjects included in the first area and one or more subjects which havebeen detected as the main subject candidates in the past and included inthe second area as the main subject candidates.
 13. The focus adjustmentapparatus according to claim 1 further comprising a display unit thatdisplays an image based on the image signals output from the imagesensor, wherein, in a case where the autofocus mode is set and theoperation unit is not operated, a first display item indicating the mainsubject selected by the selection unit is displayed on the imagedisplayed on the display unit, and in a case where the operation unit isoperated, the first display item and a second display item indicatingthe main subject candidate detected by the detection unit are displayedon the image displayed on the display unit.
 14. The focus adjustmentapparatus according to claim 13 further comprising a setting unit thatsets time for displaying the second display item since the main subjectcandidate is detected by the detection unit for each main subjectcandidate, wherein the setting unit is implemented by one or moreprocessors, circuitry or a combination thereof.
 15. An image capturingapparatus comprising: an image sensor; and a focus adjustment apparatuscomprising: a detection unit that detects one or more main subjectcandidates which are candidates for a main subject from image signalsrepeatedly output from the image sensor that performs photoelectricconversion on light that is incident through an imaging optical systembased on a predetermined first condition; a selection unit that selectsthe main subject from the one or more main subject candidates; anacquisition unit that acquires operation information of an operationunit for moving a position of a focus lens included in the imagingoptical system; and a focus adjustment unit that controls the focus lensso as to focus on the main subject selected by the selection unit in acase where an autofocus mode is set and the operation unit is notoperated, wherein the selection unit selects the main subject based onan in-focus position of the focus lens in a case where the operationunit is operated and fixes the selected main subject, and wherein eachunit is implemented by one or more processors, circuitry or acombination thereof.
 16. The image capturing apparatus according toclaim 15 further comprising the imaging optical system.
 17. The imagecapturing apparatus according to claim 15, wherein the imaging opticalsystem is detachable from the image capturing apparatus.
 18. A focusadjustment method comprising: detecting one or more main subjectcandidates which are candidates for a main subject from image signalsrepeatedly output from an image sensor that performs photoelectricconversion on light that is incident through an imaging optical systembased on a predetermined first condition; selecting the main subjectfrom the one or more main subject candidates; acquiring operationinformation of an operation unit for moving a position of a focus lensincluded in the imaging optical system; and controlling the focus lensso as to focus on the selected main subject in a case where an autofocusmode is set and the operation unit is not operated, wherein the mainsubject is selected based on an in-focus position of the focus lens andthe selected main subject is fixed in a case where the operation unit isoperated.
 19. Anon-transitory computer-readable storage medium, thestorage medium storing a program that is executable by the computer,wherein the program includes program code for causing the computer tofunction as a focus adjustment apparatus comprising: a detection unitthat detects one or more main subject candidates which are candidatesfor a main subject from image signals repeatedly output from an imagesensor that performs photoelectric conversion on light that is incidentthrough an imaging optical system based on a predetermined firstcondition; a selection unit that selects the main subject from the oneor more main subject candidates; an acquisition unit that acquiresoperation information of an operation unit for moving a position of afocus lens included in the imaging optical system; and a focusadjustment unit that controls the focus lens so as to focus on the mainsubject selected by the selection unit in a case where an autofocus modeis set and the operation unit is not operated, wherein the selectionunit selects the main subject based on an in-focus position of the focuslens and fixes the selected main subject in a case where the operationunit is operated.